Method of analyzing, displaying, organizing and responding to vital signals

ABSTRACT

A system for monitoring vital signs includes: an imaging device for acquiring video image files of a living individual; a data analysis system including a processor and memory; a computer program running in the data analysis system to automatically analyze the video images, autonomously identify an area in the images where periodic movements associated with a selected vital sign may be detected and quantified; and, an interface that outputs an electrical signal corresponding to the waveform of the selected vital sign. The system may include a Graphical User Interface, which may display a visual graph of the waveform and a single video frame or a video stream of the individual.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of each of the following ProvisionalPatent Applications filed by the present inventors: Ser. No. 62/090,729,“Optical detection of periodic movement”, filed on Dec. 11, 2014; Ser.No. 62/139,127, “Method for determining, comparing, measuring, anddisplaying phase”, filed on Mar. 27, 2015; Ser. No. 62/141,940, “Methodand system for analysis of structures and objects from spatio-temporaldata”, filed on Apr. 2, 2015; Ser. No. 62/139,110, “Adaptive arraycomparison”, filed on Apr. 14, 2015; Ser. No. 62/146,744, “Method ofanalyzing, displaying, organizing, and responding to vital signals”,filed on Apr. 13, 2015; Ser. No. 62/154,011, “Non contact optical babymonitor that senses respiration rate and respiratory waveform”, filed onApr. 28, 2015; Ser. No. 62/161,228, “Multiple region perimeter trackingand monitoring”, filed on May 13, 2015; and Ser. No. 62/209,979,“Comparative analysis of time-varying and static imagery in a field”,filed on Aug. 28, 2015, by the present inventors; the disclosures ofeach of which are incorporated herein by reference in their entirety.

This application is related to the following applications, filed on evendate herewith by the present inventors: “Non-contacting monitor forbridges and civil structures”, application Ser. No. 14/757,255 filedDec. 9, 2015 and issued as U.S. Pat. No. 9,704,266 on Jul. 11, 2017;“Apparatus and Method for of analyzing periodic motions in machinery”,application Ser. No. 14/757,245 filed Dec. 9, 2015 and published as U.S.Publication No. 20160217587 on Jul. 28, 2016; “Method of adaptive arraycomparison for the detection and characterization of periodic motion”,application Ser. No. 14/757,259 filed Dec. 9, 2015 and published as U.S.Publication No. 20160217588 on Jul. 28, 2016; the entire disclosures ofeach and every one of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The invention pertains to apparatus and methods for monitoring the vitalsigns of a patient. More specifically, the invention pertains to anon-contacting video-based analysis system to monitor vital signs suchas respiration.

Description of Related Art

Respiration rate is an important vital sign. Manual methods ofdetermining respiration rate are intermittent and have proven to beunreliable. Continuous methods have limitations; either they are notaccurate or are poorly tolerated by patients. Respiration rate is a keyindicator of ventilation. Abnormal respiration rate, either too high(tachypnea), too low (bradypnea), or absent (apnea), is a sensitiveindicator of physiologic distress that requires immediate clinicalintervention.

The most common method for respiration rate measurement is by physicalassessment, either by counting chest wall movements or by auscultationof breath sounds with a stethoscope. Many studies have shown manualmethods to be unreliable in acute care settings, especially on thegeneral care floor, where the majority of patients receive care. Even ifthey were reliable, manual methods are limited by their intermittentnature.

Two continuous methods for respiration rate monitoring are used inmultiparameter monitors, viz., thoracic impedance pneumography andcapnography monitoring.

The thoracic chest wall expands and contracts during the respiratorycycle from which respiration rate can be determined by measuring changesin electrical impedance associated with this movement. Monitoring ofrespiration rate by thoracic impedance is convenient if the patient isalready monitored for ECG, but the method is prone to inaccuratereadings due to a number of factors including: ECG electrode placement,motion artifacts, and physiologic events non-related to respiration ratethat cause chest wall movement (e.g. coughing, eating, vocalization,crying).

Continuous end tidal CO₂ monitoring with capnography is the standard ofcare in surgical settings to establish end tracheal intubation. Sinceintubated patients have a clear respiratory pattern without entrainmentof room air, it is easy for the capnometer to report the respirationrate. However, capnometers that continuously monitor ventilation fornon-intubated patients require a nasal airway cannula that draws acontinuous gas sample for spectrographic measurements within thecapnometer. Capnometry measurement of respiration rate is the mostfrequent method used by anesthesiologists. This method is sensitive tocentral, obstructive, and mixed apneas. The primary limitations ofcontinuous respiration rate monitoring by capnometry are low patienttolerance of the nasal cannula and the added nursing workload to respondto dislodged or clogged cannulas during the patient stay. In addition,any entrainment of room air by the sampling cannula can cause erroneousend-tidal values. A recent study of pediatric patients showed prematurecannula dislodgement in 14 out of the 16 patients enrolled in the study.

There is a clear demand for improved methods for respiration monitoring:For general baby monitors, no reliable product exists that can measurerespiration without a pad under the bed/child or attachment on thechild's diaper or clothing. Reliable monitors are needed, in particular,to address the so-called Sudden Infant Death Syndrome (SIDS). Accordingto the CDC, every year in the U.S., more than 4,500 infants die suddenlyof no obvious cause. A significant portion of these deaths are sleeprelated, where a perfectly healthy baby simply stops breathing inhis/her sleep without warning. Other sleep related disorders affectlarge numbers of people. For example, more than 15% of adolescentscomplain of some form of sleep problems, many of which may contribute tomisdiagnosis of ADHD and other behavioral problems.

Improved monitoring is needed in the hospital environment as well. Inspite of its clinical importance, respiration rate is the last corevital sign without a reliable and continuous monitoring solution thatpatients can easily tolerate. The lack of a reliable respiration ratemeasurement is a major contributor to avoidable adverse events. Oneretrospective study of over 14,000 cardiopulmonary arrests in acute carehospitals showed 44% were of respiratory origin. Another study reportedthat respiratory failure, a key Patient Safety Indicator (PSI), hasincreased in U.S. Acute Care Hospitals. The reported incidence is 17.4per 1,000 hospital admissions, leading to over 15,000 avoidable deathsat a cost to the healthcare system of over $1.8 billion.

OBJECTS AND ADVANTAGES

Objects of the present invention include the following: providing animproved system and method for measuring respiration rate; providing anon-contacting respiration monitor that is reliable under normalsleeping conditions; providing a stand-off monitor for vital signs;providing a method to extract respiration rate from a video file;providing a graphical user interface that displays respiration data;providing a user interface that directly associates respiration with avideo stream so that selected events may be time-stamped and associatedwith corresponding frames of the video image; and providing an improvedtool for detecting sleep apnea, analyzing different types of apnea, andwarning of SIDS-related events in real time. These and other objects andadvantages of the invention will become apparent from consideration ofthe following specification, read in conjunction with the drawings.

SUMMARY OF THE INVENTION

According to one aspect of the invention, a system for monitoring vitalsigns comprises:

a device for acquiring video image files;

a data analysis system including a processor and memory;

a computer program to automatically analyze the video images, identifyan area in the images where periodic movements associated with aselected vital sign may be detected and quantified; and,

a user interface in which the temporal variation of at least one vitalsign may be displayed along with at least one video frame correspondingto a selected time in the temporal variation display.

According to another aspect of the invention, a system for monitoringvital signs comprises:

a device for acquiring video image files;

a data analysis system including a processor and memory;

a computer program to automatically analyze the video images, identifyan area in the images where periodic movements associated with aselected vital sign may be detected and quantified; and,

an interface that outputs an electrical signal corresponding to thewaveform of the selected vital sign.

According to another aspect of the invention, a system for monitoringvital signs comprises:

a device for acquiring video image files;

a data analysis system including a processor and memory;

a computer program to automatically analyze the video images, identifyan area in the images where periodic movements associated with aselected vital sign may be detected, and quantify the rate of periodicmovements using an adaptive array comparison method; and,

a user interface in which the temporal variation of at least one vitalsign may be displayed.

According to another aspect of the invention, a method for monitoringvital signs comprises the steps of:

acquiring video image files of an individual;

in a data analysis system including a processor and memory, using acomputer program to:

-   -   automatically analyze the video images, autonomously identify an        area in the images where periodic movements    -   associated with a selected vital sign may be detected, and    -   quantify the rate of periodic movements using an adaptive array        comparison method; and,

displaying the temporal variation of at least one vital sign on a userinterface.

According to another aspect of the invention, a method for monitoringvital signs comprises the steps of:

acquiring video image files of an individual;

using a computer program in a data analysis system including a processorand memory, to:

-   -   automatically analyze the video images,    -   autonomously identify an area in the images where periodic        movements associated with a selected vital sign may be detected,    -   quantify the rate of periodic movements using an adaptive array        comparison method, and,    -   store the quantified data and the video image file with a common        time log so that particular changes in periodicity of the vital        sign may be associated with corresponding video frames; and,

displaying the temporal variation of at least one vital sign on a userinterface.

According to another aspect of the invention, a system for monitoringvital signs comprises:

a device for acquiring video image files;

a data analysis system including a processor and memory;

a computer program to automatically analyze the video images,autonomously identify an area in the images within a preselectedperimeter where periodic movements associated with a selected vital signmay be detected, and quantify the rate of periodic movements using anadaptive array comparison method; and,

a user interface in which the temporal variation of at least one vitalsign may be displayed.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the invention. A clearerconception of the invention, and of the components and operation ofsystems provided with the invention, will become more readily apparentby referring to the exemplary, and therefore non-limiting embodimentsillustrated in the drawing figures, wherein like numerals (if they occurin more than one view) designate the same elements. The features in thedrawings are not necessarily drawn to scale.

FIG. 1 illustrates schematically the arrangement of video data into athree-dimensional array where x, y are spatial coordinates in a videoimage and z is time.

FIG. 2 illustrates the result when the frame spacing is non-optimal, inthis case, every 8^(th) frame (N=8).

FIG. 3 illustrates the result when the spacing is more nearly optimal,in this case, every 6^(th) frame (N=6).

FIG. 4 illustrates a summed frame of differenced frames from 20 secondsof video data. Note the method isolated motions associated with thebreathing based on the selected frame number separation as indicated bythe darkest pixels in the summed frame.

FIG. 5 illustrates a simplified flow chart of the basic camera operationand analysis done in a completely automated process.

FIG. 6 illustrates the spectral response of a typical color video sensorfor the three colors red, blue, and green.

FIG. 7 illustrates typical responses for R, G, B, and IR sensors.

FIG. 8 illustrates the spectral response for several commercial thermalsensors [Thermal Sensors for General Applications—S3xxC Series,Thorlabs, Newton, N.J.].

FIG. 9A illustrates a regular or healthy respiration waveform, and FIG.9B illustrates an irregular and transient event in an otherwise periodicmotion.

FIGS. 10A-10D illustrate the steps in an exemplary analysis for thespecific case of respiration, wherein 10A shows 4-frame spacing, 10Bshows 8-frame spacing, 10C shows 9-frame spacing, and 10D shows 9-framespacing but with a 5-frame offset.

FIG. 11 illustrates the ability of a user to rewind the acquired datafiles (indicated schematically by the large arrow) to return to a pointin time at which an event occurred (here, an irregularity in therespiration waveform).

FIG. 12 illustrates the respiration of an infant in which breathingstopped for a short period.

FIG. 13 illustrates an example of a user interface implemented formobile devices.

FIG. 14 illustrates a respiration waveform containing patterns that maybe associated with particular pathologies (arrow).

FIG. 15 illustrates the successful capture of individual respiratorywaveforms from mother (Individual 1) and child (Individual 2) sleepingtogether, using the present invention.

FIG. 16 illustrates the simultaneous analysis of breathing via IR (leftpart of image and upper data trace) and visible chest displacement(right part of image and lower data trace).

FIG. 17 illustrates one example of a user interface displaying videoimage(s) along with data derived by the inventive methods.

FIG. 18 illustrates the different sleeping respiration rates of twomales of the same age.

FIG. 19 illustrates a scene in which the perimeter of a user-definedmonitored region is shown as a heavy white line.

FIG. 20 indicates several possible examples of sources of movement(arrows) that might be detected by traverse of the defined boundary.

FIG. 21 shows the placement of boundaries (heavy white lines) to definemultiple monitored regions in a single video image of a neonatal careenvironment.

FIG. 22 shows some possible sources of repetitive movements in a room inwhich two individuals are sleeping and are to be simultaneouslymonitored for respiration.

FIG. 23 shows a simple rectangular boundary substantially defining thearea inside an infant's crib.

FIG. 24 shows the use of two perimeters surrounding an interior regionof interest.

FIG. 25 shows the use of multiple perimeter regions to help identifymovement of objects into or out of the perimeter or the interior region.

FIG. 26 shows the use of a user-defined perimeter in an outdoorenvironment.

FIG. 27 illustrates schematically one sequence of steps for determiningand displaying phase in accordance with one aspect of the invention.

FIG. 28 illustrates schematically one configuration to interface theinvention with existing sleep center hardware and methods.

DETAILED DESCRIPTION OF THE INVENTION

In its most general sense, the invention comprises a video-based systemfor observing a patient, and using a processor to: first, identify areasof a video image that represent periodic movement associated with avital sign, such as respiration or pulse rate, second, to quantify andtrack the periodic movement, and third, to output data related to theperiodic movement through a selected user interface. The user interfacemay be a dedicated interface, for instance, to use in a hospital oracute care setting. Alternatively it may consist of a softwareapplication configured to operate on a mobile device, for instance, if aparent or caregiver needs to monitor a sleeping infant. The data streammay be presented in real time as a simple display of respiration rate,pulse rate, etc., or it may be archived for later review. Archived dataare preferably time-stamped so that any selected part of the deriveddata may be associated with the respective video frame(s) correspondingto the selected time. A user interface may allow the user to displaydata for a selected time interval and then select a particular timewithin that interval, for which the corresponding video frame will bedisplayed. Alternatively, an interface may output a signal thatrepresents the respiration waveform. The output signal may be in anyconvenient analog or digital format, e.g., 0-5 V, 4-20 mA, and may bepart of a network, wireless network, mesh network, or other control andautomation system operating on any convenient protocol, e.g, HART,WirelessHART, ZigBee, IEEE 802.15.4, etc.

It will be appreciated that many “video cameras” and “video recordings”include not only images but also the corresponding audio data,synchronized with the image data. The invention can make use of theassociated audio data in a number of ways, as will be described inseveral Examples.

It will be appreciated that the term “patient” or “individual” is usedherein for convenience, and is intended to cover any human or animalthat is to be monitored for vital signs. The term “patient” does notnecessarily imply that the individual is ill or is presently undergoingmedical treatment. Some non-limiting examples include: a sleeping infantbeing monitored for sleep apnea, SIDS, or other signs of distress; apatient in a hospital, emergency room, or nursing home; a patientundergoing study for sleeping disorders; a soldier in a combatsituation; a person in a crowd being monitored for signs of stress orcommunicable disease; or an animal under veterinary care.

The invention may be conveniently integrated into currently availablevideo monitors to derive real-time information such as respiration rate,cessation of breathing, respiration waveform, and motion events. It canalso study more than one individual in a video file and separate theirindividual waveforms for analysis. It can also be easily integrated withany camera that has infrared technology and recording capability. Theuser interface may display real-time information (video and graphs) todisplay respiration rate, cessation of breathing events, the actualrespiration waveform, and motion events. The data, and associated videoframe(s) can provide visual feedback on the quality of a patient'ssleep, while identifying potential issues such as central andobstructive sleep apnea, breathing cessation, and sleep inhibitors. Itcan also be used for post-surgery assessment of the impact on sleep.Further, the data may be conveniently structured to produce reports andvisuals that better inform parents of potential sleep related issuesthat need to be addressed during visits to the pediatrician.

In the examples that follow, various aspects of the invention will bemade clearer, and applications to various monitoring problems will beillustrated. These examples are not intended to restrict the scope ofthe invention to the particular implementations described. In someexemplary renderings of the user interface, the identity of the patienthas been obscured for the purposes of inclusion in this disclosure;however, it will be understood that in an actual clinical setting theface of the patient would typically be visible to some degree in thevideo images and not obscured or blacked out as it is herein.

Method of Adaptive Array Comparison

Example

One way to analyze a video stream to extract waveform information isAdaptive Array Comparison, which may be described generally as follows:A video signal includes multiple frames separated in time and each frameincludes multiple pixels. Each pixel value in a particular frame iscompared to the values of corresponding pixels in different frames ofthe video signal to search for periodic signals occurring at the pixellocations over time. Periodic signals are defined in time with a maximumand minimum peaks. These two locations give the maximized absolute valuein the difference in two locations of the periodic signals. This factcan be exploited to filter and locate pixel locations where the periodicsignal gives the largest signal to background or signal-to-noise in ascene or image sequence. In the case of human respiration we know theadult rate typically lies in the approximate range of 12-20 breaths perminute. This corresponds to 0.25 to 0.33 Hz. A medium rate would be 16breaths per minute or 0.27 Hz. For a 15 fps camera that corresponds to abreath every 56.25 frames. This would tell us that a video of a personbreathing may include periodic signals at certain pixels having amaximum and minimum at a difference of approximately 28 frames apart. Sothis could be the initial frame difference the system might use tolocate the best pixels to track. The system will difference severalseries of images at this separation, meaning the value at each pixellocation in one frame is subtracted from the value at the correspondingpixel location at a second frame. Then, the second frame is differencedwith a third frame and so on. For each pixel location, the absolutevalues of their differences are added together. Then some number ofpixels with the highest sum of differences are selected to be tracked.There is the potential that the chosen frames happen to be 90 degrees(or some other phase shift) out of phase with a max and min; so in theevent that no initial peaks are found, the system will recalculate witha 90 degree phase shift (or some other phase shift). Once it has foundthe correct pixels to track, it will then begin peak counting for eachselected pixel, noting the phase of the waveform. Once it hassufficiently determined the precise phase and frequency of the waveform,it will recalibrate, making sure to frame difference such that thenumber of frames between differences exactly equals the differencebetween a max and min of the waveform, as well as starting in phase sothe difference is aligned with the max and min. The processor runs thecode to do this like any other programs, although a specialized piece ofhardware specifically designed to do this is not necessarily used (incontrast to the way decoding HD video is typically done).

Applicant has found that the foregoing method adapts to each person'swaveform even if it changes. In the example of breathing, differentfrequency rates can be chosen to start with, based on the age of theindividual or previously stored user data. This method inherentlyfilters unwanted frequencies. Because of the selected time differencebetween frames we reject signals associated with frequencies other thanthose related to respiration.

Example

An image sequence is obtained through a video camera and stored intomemory. A video sequence as shown in FIG. 1 has dimensions of space inthe x and y axis while having the dimension of time in the z axis andcan be thought of as a 3-D data space. The video sequence contains aperiodic or recurring motion of interest that is to be extracted fromthe data set. For example, a feature of interest may be occurring at agiven pixel and we may be interested in monitoring that feature. Thesystem can use its temporal behavior to find and locate that pixelautonomously without any knowledge of the scene, the surrounding pixels,the spatial characteristic of the local pixels or where that pixel maybe.

Method of Adaptive Array Comparison for the Detection andCharacterization of Periodic Motion

Example

Mathematical Description:

An image frame is defined as an X-Y matrix. A video file is a set ofimage frames defining an X-Y-t matrix.

For each pixel (i,j) one can calculate the difference (D_(i,j))^(M,N)between the value at pixel (i,j) in one frame and the value in anotherframe, where M is the starting frame number and N is the spacing betweenthe two frames. So (D_(2,3))^(4,9) would be the difference in value orintensity at pixel (2,3) in frame 4 compared to that in frame 13.

The difference matrix is then summed (preferably in quadrature) to yieldthe total difference in pixel intensities between frames M and M+N.Difference matrices are calculated for various values of M and N, tofind M and N that give the highest value of summed differences. Then aselected number of pixels (i,j) having the greatest difference arechosen and their intensities are tracked over time to determine theperiodicity of movement.

There is generally a limit on how long one does this. For example, at 15fps for 20 sec there are 300 frames, so if N is 10 the system woulddifference 29 times (accounting for the ends) or less as M is increased.

Once this is done initially and the system has found the location of apeak and the peak separation it will redo the calibration with aspecific M and N to get it exactly on the peak. M would now be the framenumber where an expected max or min occurs and N would be the value of ½of a waveform. This introduces the novel aspect that the process becomesessentially adaptive.

Note that in the preceding Example, the difference matrix was summed inquadrature. Applicants recognize that this is only one of several waysthat the values may be combined, such as absolute difference, where thecombination of differenced frames has a cumulative effect that increaseswith more of the desired signal, in this reference motion.

Example

The method and functions of Adaptive Array Comparison may be describedas follows:

-   -   1. Video Sequence comes in at some frames per second (fps)    -   2. Some seconds of that data is continually buffered. (Previous        frames are overwritten with new frames.) These are the frames        that will be processed.    -   3. From the buffered frames [1], [2], . . . [n] a multiple of        frame differences N to calculate is selected, e.g. every 4th        frame (N=4) or every 5^(th) frame (N=5), etc. This allows the        program to find the best periodic motion rate. The time range        between frames is selected based on the range of periodic motion        we are interested in finding. This also acts as a band pass        filter giving preference to the periodic motion rate within this        range.    -   4. The program also offsets these frames, say, starting with the        1^(st) frame, then the second, etc. This allows it to find the        best phase. So for example if it is subtracting every 4^(th)        frame, it first would do the 1^(st) frame minus the 5^(th), the        5^(th) minus the 9^(th), and so on. Then it would do the 2^(nd)        frame minus the 6^(th) frame, then the 6^(th) frame minus the        10^(th).    -   5. For each test, frame differencing is conducted for some        number of frames. For example, 11 frames may be used and 10        frame differences will be calculated. Each frame difference will        be an array of absolute value numbers with each position in the        array corresponding to a pixel location.    -   6. After 10 frame differences are calculated the square of the        frame difference arrays are added together to produce a total        frame difference array. Then the total frame difference array is        analyzed to find the array locations having the largest values.        For example, the ten largest values in the array may be found        and the array locations containing those values are recorded.        When subtracting the frames the program adds all the differenced        frames from the buffered video in quadrature, meaning it will        square the difference values so that negative and positive        differences don't cancel. Motion in opposite direction shows up        in the difference frame with an opposite sign but for the        present purposes it needs to add positively, otherwise back and        forth motion can show as zero in the sum of the differenced        frames.    -   7. From all the total frame difference arrays across all the        multiples and offsets the program finds a selected number of        pixels that have the largest values, say, the brightest 10        pixels. These pixels represent the largest periodic motions in        the scene that are between the rate boundaries set in Step 3.    -   8. Those pixels' values are tracked over time and this will plot        the motion waveform. The signals from the tracked pixels are not        necessarily in phase. Any two signals could be 180 degrees out        of phase and still provide the same information about the        existence and the frequency of the same time periodic signal.        For example, the signals of all pixels could be out of phase        with all other signals, but still correspond to breathing of a        subject. Thus the frequency of each signal would be the same.    -   9. From the motion waveform the system will determine the rate        and phase. In terms of frame differencing this translates to the        peaks in the waveform occurring every N^(th) frame and the        waveform starts at frame M.    -   10. From the information in Step 9 the Frame Differencing method        adapts since it knows where the peaks start and how many frames        apart they are.    -   11. Now it subtracts frames with a separation exactly equal to        the separation between a maximum and minimum in the waveform. It        will also make sure to start this process exactly on a peak or        minimum. This optimizes the rate and phase to precisely select        the motion waveform.    -   12. This process can be repeated based on a number of factors:    -   a. Time—The method can reiterate every n seconds to ensure it is        optimized for the existing waveform.    -   b. When there is a large excursion in the measured waveform the        process can restart as this can be due to a motion event.    -   c. The process can restart based on a trigger from an external        motion detection method.    -   d. The process can be restarted on buffered data (the 20 seconds        of previous video) when an alarm is triggered, (for example, no        peaks are detected) to ensure the alarm is accurate. For        example, if the waveform is accidentally lost this step could        check the last 20 seconds of data to see if the waveform can be        detected in another pixel or set of pixels.

It is important to emphasize that the process described above isperformed autonomously by the system software without the need for userintervention.

Example

FIGS. 2 and 3 illustrate how the frame spacing is selected by trying twodifferent spacings and comparing the magnitude of the peak differences.In FIG. 2, every 8^(th) frame is used; this turns out to be a bad or“null” spacing, as the sum of differenced frames for a given pixel iszero. When the spacing is changed to every 6^(th) frame, the sum ofdifferenced frames for a given pixel is 30, indicating a much highersignal for this waveform that distinguishes it from the rest of thepixels.

So therefore, the program would try a multiple of spacings thatrepresent a reasonable range of breathing rates. The program would alsooffset them or change the phase because the best spacing that perfectlyfinds the peaks and valleys is also a “null set”. Now it knows where tolook when this process yields the pixel with the largest sum of thedifferences. From this pixel it tracks and locates the peaks and valleysof the waveform. Then it will adapt to the individual's waveform withthe next calibration.

Example

FIG. 4 illustrates the summed frame of differenced frames from 20seconds of video data collected on a human subject. Note the method hassuccessfully isolated motions associated with the breathing based on theselected frame number separation: here the darkest pixels identify thelocation of greatest motion found in the chest region from the up/downright/left motion of breathing as seen by the camera. These are thepixels that would then be tracked to determine the breathing waveform.It is important to emphasize that the data presentation in FIG. 4 is notan image per se, but rather simply a graphical display of the pixels(dark) that show the most movement.

Example

An explicit example of the calculations can be shown as follows: Let [1]be the first frame of a video sequence, a 640×480 image, so that [1] isa 640×480 array. Likewise [2] would be the second frame making up a new640×480 array and [3] would be the third. We would like to sum thedifference of every N frames. To ensure the difference is positive wesubtract the frames, then square the difference, and then take thesquare root. Finally we sum all of differenced frames.

For example, if we difference every 8 frames the calculation would be ofthe form:√{square root over (([1]−[9])²)}+√{square root over(([9]−[17])²)}+√{square root over (([17]−[25])²)}+√{square root over(([25]−[33])²)}=[SUM]

Other potential applications and features of the invention include thefollowing:

A user defined setting can be selected (e.g., age) to narrow the windowin which rates are expected. Infants, for example, breathe much fasterthan adults and a low rate is unlikely. It will be appreciated thatnarrowing the window allows the system to converge more quickly on anoptimal frame rate, because this reduces the number of iterations thesystem has to go through, making it quicker and more accurate as thechance of error would be reduced by eliminating certain frequencies.

Information such as a profile of a particular user's sleepingrespiration rate, may be stored and later retrieved so the device has abetter range of expected rates to look for a priori. In this case theuser selects a profile that the device has gathered over previous usesor parameters that were previously entered.

Sections of the video scene may be selected to narrow the search. Forexample, if a baby monitor is looking at crib, the user interface mightallow the user to draw a box (e.g., on a touch screen) to select onlythe crib, eliminating the rest of the room. An adult may select his/herbed. Eliminating extraneous parts of the image from consideration willallow the calculations and optimization to proceed more quickly.

A user may draw a line down the middle of the bed designating areaswhere two people sleep so each person can be tracked individually. Otherapproaches may not require a user to select or “draw” a region ofinterest. For example, a vicinity of pixelated focal plane or in a fieldof view area may be designated by proximity to a bright pixel, portionof an area, or other feature.

One can isolate periodic motion by selecting the range the motion isexpected to be in. For example, when studying heart rate or pulse, areasonable value might be from 60 to 120 bpm.

The system may also be used to determine if something has gone out ofrange and/or activate an alarm if, for example, no breathing or a heartrate can be found in the expected range. The system may enter acalibration state prior to alarming, to check for a false positive on analarm. The system may use data in memory, e.g., the last 20 seconds, toinstantly analyze this prior data to locate a signal elsewhere in thescene. If no such signal exists, the system may continue with the alarm.If such a signal exists the system may not enter an alarm state andinstead indicate that a signal has been found.

The data can be used with a standard peak finding method to determinethe max and mins of the waveform.

The system can be used to determine a person's presence, for example theexistence of vital signs in a controlled space where no one is supposedto be.

Example

FIG. 5 presents a simplified flow chart of the basic camera operationand analysis done in a completely automated process. The input video maybe MPEG1 simple profile or MJPEG from, e.g., a USB webcam or similarsource. The initial calibration subroutine, typically using 20 s ofvideo, locates the 5 pixels with the greatest values in differencematrix, and establishes State 0, or initial calibration. The waveformtracking and processing subroutine tracks pixels, continuously outputsthe last N values of the pixels determined to be the greatest from thedifference matrix; processing is done on waveform to determine outputstates. States 1-3 will be determined in this routine based onprocessing of the waveforms. An ongoing calibration subroutine iscontinuously looped; this uses 60 s of frames, summing the difference offrames and locating the five pixels with greatest values in differencematrix. Five output states are continually outputted from thesubroutines through a physical channel. Off State condition may bedetermined by a selected trigger, implemented either in hardware orsoftware.

Example

Calibration Subroutine

A subroutine recalibrates the location to find the best pixels. Thisconsists of going through the process of frame differencing again andlocating the 15 highest valued pixels in the summed array. The durationof the calibration can be programmatically controlled as well as theframe numbers to difference as a result they may vary. For example wechoose to difference every 4 frames of a 16 fps camera for 40 secondsresulting in 160 differenced frames. Note this is different than theinitial calibration since it is limited to 20 seconds.

This recalibration process continually goes on in the background whilethe waveforms are being outputted from the pixels and the peak findingis performed.

It is important to keep in mind that the mathematical techniques of thepresent invention derive parametric outputs whether or not an image isever created or portrayed. Thus, techniques of the present invention maybe used with monitors that do not display images, do not recognizeobjects, and do not have frequent human observation or similarparticipation. For example, the present invention may output a commonlyreported characteristic such as a breathing rate or heart pulse rate orphase or a lag interval or a dimension of a periodically moving objector a period of a motion, or a timing or a motion or other informationassociated with a periodic motion event without displaying an imagethereof. Conversely, in some examples, the user interface may includeactual video images, which may be selected to correspond to a particularpoint in time when an output parameter has a particular value or thewaveform displays a particular feature (e.g., an episode when breathingchanged or stopped temporarily).

As used herein, the term “video” describes any data set representing aseries of digital (i.e., pixelated) images of a scene, taken at fixedtime intervals, so that the data represent points in X-Y-t space. Theimage may represent a pattern of reflected and/or emitted visible light,UV, IR, X-rays, gamma rays, or other electromagnetic radiation detectedby a two-dimensional position-sensitive detector. Although certainstandard file types have been described in some of the examples, theinvention is not limited to any particular frame rate or file format.

It will be further appreciated that the invention is not limited to anyparticular type of image acquisition hardware; video cameras, webcams,digital cameras integral in cell phones, etc., may also be used togenerate the raw data files. The digital imaging device may furtherinclude any suitable lenses or other optical components, such astelescopes, microscopes, etc., as are well known in the art. Inparticular, the invention is well suited for examining periodic movementin small biologic systems, such as heart contractions in an embryo,which could be observed using a video microscope. Adapted to atelescope, the invention could be used, e.g., to study periodicphenomena occurring on the surface of the sun.

Many examples of the present invention are completely general in thatthey do not require or insist upon a blob that must be identified withan object in the field of view or with a contour segment that must beassociated with an object in the field of view.

Techniques of the present invention may be applied to a variety ofimaging, including visible imaging, thermal imaging, multispectralimaging, or hyperspectral imaging. In fact these are entirely differentand independent media having different sources and different mechanismsand different physical significances. However, the techniques formeasuring motion remain the same for any spectral ranges. For example,the use of visible images of an object of interest overlaid (orinterleaved, overlapped, interspersed, approximately synchronized, ortruly simultaneous) with near or far infrared images may yield twoeffectively independent views of an object of interest. If reflectedvisible light reveals a periodic motion which may be associated with abreathing or a pulse or a structural vibration or some other periodicmotion, and a thermal image reveals a similar periodic motion locationproximate to the visible finding and similar in phase, frequency, oramplitude, or all three, then this improves the likelihood of anaccurate result rather than a false positive or false negative finding.

As noted above, the imaging systems may have multiple inputs. These maycomprise two visible cameras, an infrared imager and a visible camera, acamera and another input other than an imager such as an ultrasonicsensor or a temperature or a pulse monitor or some other combination oftwo or more imaging devices.

The inventive technique is not limited to a particular wavelength oflight. Different colors are represented by different wavelengths oflight, e.g. 550 nm is green. Amplitude changes that are detected by thistechnique can be restricted to a single wavelength of light or representa summed intensity change over multiple wavelengths. FIG. 6 shows theresponse of a typical color camera. Each wavelength can be measuredindependently or together (mono grayscale). The inventive technique may,for example, monitor only the green, blue or red wavelength or monitorthe sum of all three.

Electromagnetic Wavelength options. In addition the inventive techniqueis not just limited to visible wavelength of light, but can be used inthe near IR, far IR, or UV. The technique could be extended to anysensor type that can measure changes in light levels over time whetherfrom reflective or emissive sources. Visible light is generally althoughnot always measured as a reflection. Thermal IR light is generally butnot always an emission from the surface. The invention works regardlessof whether or not the target is reflecting or emitting the light.

Sensor selection. The sensor type can be chosen based on the scene ortarget. For example, if the scene is completely dark, void of a visiblelight source, a thermal IR sensor may be used to monitor the changes inlight levels. Also if a particular material or substance is the targetand light level changes are due to a property of interest on the targetanother sensor type may be chosen. For example, with gas that absorbs incertain wavelengths, or more generally chemicals, a particular sensorthat detects those properties may be chosen. For example, one may beinterested in using the technique to find an exhaust or chemical leak inthe scene based on light intensity changes from the leak specificallyassociated with the absorption and/or emission at certain wavelengths.Another example may be blood flow that absorbs in certain colors, andthat flow changes or pulsing may be indicated by intensity changes in acertain wavelength of light, then a sensor particularly sensitive tothat wavelength of light might be chosen.

Interpreting measurement information. The inventive technique can alsobe used to garner information about the type of change. A particularchange using a thermal sensor would indicate that the temperature ischanging, whereas a change in color may indicate the target is changingis absorption or emission properties. A change in amplitude could alsobe indicative in a change in position or vibration, whereas a change inposition of the signal being detected from pixel to pixel in time maygive information about displacement.

Comparing multiple measurements. Ratio or comparisons of color changesor amplitudes of certain wavelength can also be used. For example, itmay be useful to locate a pixel that changes in intensity from blue tored. This could be indicative of certain properties of interest. Anexample would be pulsing of blood. The technique could be used to locatea pixel of interest that is indicative of blood flow so that parametercan be tracked. Multiple sensors could be used for this technique or asingle sensor with wavelength filters applied (such as a typical colorcamera). Certain features of interest may be indicated by relationshipsbetween multiple sensor sensitivities or wavelength of light.

Redundant and independent inputs. Multiple sensor types or wavelengthdetections could also provide multiple detections of the same phenomenonincreasing the confidence of detection. For example, the light intensitychanges due to the periodic motion of the chest from breathing may bedetected with a visible or IR camera pointed at the chest while anothersensor looks at the intensity change in thermal IR from temperaturechanges around the nostril indicative of inhalation and exhalation. Thetechnique is then use in both cases to strengthen the detection scheme.

False negative findings. Multiple wavelengths could be used to discernor improve findings which may be false positive and false negativefindings and true positive and true negative findings. Intensity shiftfrom multiple wavelength, red, blue, green, IR, etc. could be used inconjunction with each other to improve the signal to noise ratio andalso provide multiple independent readings to increase confidence indetection.

FIGS. 6 through 8 all show the way in which different sensor typesdetect different wavelengths of light. Specifically, FIG. 8 shows how aparticular wavelength of light is associated with a physical phenomenon,in this case detection of light in different spectral ranges by severaldifferent thermal sensors.

Measurement duration. This technique could be used with signals that arerepetitive but only over a short time duration. The technique could beapplied to shortened windows of time to catch signals that occur onlyfor a set duration. Furthermore it could be used for signals thatcontinually change over time but are ongoing. For example, with a signalthat speeds or slows, the time that is used to calibrate or search for acertain intensity change could be shortened to be less than the timechange of the signal itself.

Transient event. Additionally there may be irregular or transient eventsin a periodic signal. This technique could be used in a short enoughtime window or in a sufficient sequence of waves to extract the locationof a periodic signal in the presence of irregular or transient events.FIG. 9 shows an irregular and transient event in an otherwise periodicmotion. If the sample window for the technique described here isproperly placed the maximum and minimum of the periodic signal can belocated. Multiple phase offset would help to address this issue bybuilding up a pixel's sum of differences at a time that the phase offsetfor a starting point has brought it past the irregular or transientsignal occurrence.

Spatial proximity. This technique can find multiple pixels of interest.Spatial relationships between the pixels of interest can further beexploited. For example, if a larger number of pixels of interest wereselected and the vast majority of them were found to be in closeproximity to each other that could indicate those pixels are related tothe same physical phenomenon. Conversely, if they are spread apart andthere appears to be no spatial coherence or statistical meaning amongthe spatial relationship or they are randomly spaced that could indicatethey are not related. Furthermore, this technique could also be used toincrease confidence in the signal or improve findings which may be falsepositive and false negative findings and true positive and true negativefindings. For example, in a respiration there are likely to be manypixels of interest found near the chest. We could expect a certainpercentage to be heavily localized. If this is not the case it may lowerour confidence that the respiration was detected. Conversely, if a largenumber are heavily centralized we may be more confident we have locateda physical region undergoing motion from breathing. The confidence maybe set by a weighted spatial location mean of the pixels or averageseparation distance, or standard deviation from the spatially averagedlocations of the all pixels of interest.

Breaths per minute. Intensity variations for different pixels ofinterest can be indicative of certain phenomena of interest. By limitingthe temporal separation of which the pixels are differenced and thedifferenced sum is obtained we can filter for phenomena of interest. Forexample if we are interested in breathing we limit our frame separationto max and min separation time of waveforms that are indicative ofbreathing, say 10 to 30 breaths per minute, whereas for the blood pulsewe may limit the difference based on max and min separations that fallin the 50 to 100 beats per minute range.

Re-calibration—finding a pixel of interest. It is possible after thetechnique adapts to find the suitable or best separation to get thelargest intensity change based on the differencing of max and minframes, a new search can be performed with that knowledge with tighterconstraints to search out specifically that waveform. In that sense itis adaptive after it uses more liberal parameters to find the initialsignal. It is possible that a user's information or information on asubject or phenomenon may be stored. The technique can now be used witha priori knowledge of rate, phase etc. to speed up finding the pixels ofinterest. For example, a user may store his profile, and the techniqueis then used with knowledge of that user and his typical breathing rate.That way, fewer cycles need to be performed and a tighter constraint canbe placed on the technique to find the pixel of interest. For example,only a certain separation of frames are used based on the breathing rateand only the different phases are cycled through.

Visible and infrared photons. Variation in the intensity of pixels maynot always result from radiation emitted or reflected by a singleobject. For example, if something is moving and at a differenttemperature than the background, that object may move back and forthperiodically blocking a small portion of the background. To a thermalsensor, a pixel detecting light in that region will see an increase anddecrease in brightness from the object moving back and forth as theobject at T₁ and then the background at a different temperature T₂ arealternately imaged by the pixel.

Multiple cameras. Multiple cameras can be used for multiple detectionschemes. They potentially could run at different rates. It is possibleto temporally align frames so that certain frames occur at the sametime. In this scene the resulting detection of a signal can betemporally aligned as well and correlated. Cameras could potentially beset to image the same field of view. Pixels across multiple cameras orsensors could be correlated so spatial relationships of the pixels inthe image of each camera is known.

Other sensors. Other inputs could be correlated to one of more cameras.The detected signal could potentially be correlated to another inputsignal as well. For example, if a pulse oximeter provides input to thesystem, the blood pulse and potential respiration timing could be usedto validate or increase the confidence of a detected signal determinedfrom a pixel of interest from the technique. Tachometers,accelerometers, and tonometers are all examples of types of sensors thatcould be used in conjunction with the inventive technique. These inputsignals could also provide frequencies or phase data to allow the systemto use tighter constraints to reduce the number of iterations it goesthrough or immediately determine the proper phase and or frequency fromwhich to select the differenced frames. These inputs also can be used astriggers for recalibration or other functions.

Single pixel and combination of many pixels Techniques of the presentinvention may be used with the smallest achievable pixel size or may beused with binned pixels where multiple neighboring pixels arecollectively associated (mathematically or statistically) to create alarger virtual pixel. This technique may be done on camera or chip ordone afterwards in data processing. Binning may be done horizontally,vertically, or both, and may be done proportionately in both directionsor disproportionately. Collective association or binning may potentiallyleave out or skip individual pixels or groups of pixels. For example,one form of collective association may comprise combining a plurality ofbright pixels while ignoring some of all of the pixels not determined tobe “bright” or “strong” considering a characteristic of interest such asa selected frequency range or amplitude.

Gaining confidence by eliminating false findings. It may be of interestto increase the confidence of the detection by exploring neighboringpixels. Even if those pixels were not chosen as the ones exhibiting thelargest motion they can be explored to determine if at least one or moreexhibit the same or strongly correlated waveforms to the pixel ofinterest. If a physical phenomenon that one is trying to detect isexpected to be larger than one pixel, it stands to reason thatneighboring pixels undergo similar behavior. Thus it will be clear thatthis could be used to eliminate false positives in a detection scheme.

Multiplexing. The inventive technique can be applied in a single pixelvariant in which an optical element would be used in a multiplex modewhere the optical element scans the scene and the transducer samples ateach point in the scene. An image or array is created from the scanneddata. The sampling/scanning rate is contemplated to be high enough toeffectively sample the scene at a fast enough rate to see time-varyingsignals of interest. Once certain pixels of interest are located, thesystem would then need only scan those pixels until a recalibration isneeded.

Searching a plurality of frequencies. One can compare amplitudes ofdifferent subtracted frames separation values, or multiple sums ofsubtracted frames separation values. For example, comparison can be madebetween the sum of the subtracted frames for separation X and forseparation Y. The frame separations are indicative of frequencies. Thiscomparison will allow one to compare amplitudes of signal changes fordifferent frequencies. Multiple frames separation values that giveinformation about amplitudes of a frequency of the signal can be used toconstruct a frequency spectrum for a single pixel or multiple pixels.

Arrays representing subtracted frames or sums of subtracted frames atcertain frame separation values may be indicative of a frequency. Thosearrays may be compared to indicate information about the signals. Forexample, if two arrays are determined that are indicative of frequencyf₁ and f₂, we may compare those two arrays for determine the spatialdistance between the phenomenon that is causing the frequencies. In thiscase the array may be a spatial image.

The following example will more fully illustrate the inventive method,applied specifically to the case of monitoring respiration.

Example

Initial calibration with a single frame separation value and startingpoint for frame differencing does not optimize the differenced valuesspecific to the respiration rate or maximum and minimum values in thechest motion. To solve this the system will select multiple frameseparation values, all at multiple starting points, to ensure that itfinds the optimized signal of interest. A series of waveforms, FIGS.10A-10D demonstrates this principle.

Here we see that at 4 frames of separation, FIG. 10A, the separationdoes not align with the maximum and minimum peaks in the waveform.Aligning with the maximum and minimum peaks would give the strongestsignal indicating that the right separation or rate has been found.

FIG. 10B shows the effect of changing the separation between differencedframes to every 8 frames. One can see that this is better but not quiteoptimal. Next consider 9 frames, FIG. 10C. To ensure that allpossibilities are considered we want the option to select a range forframe separations to subtract as well as the increments in spacing. Forexample, we subtract from every 2 to 30 frames in increments of 4, orgenerally we subtract from N₁ to N_(n) in frame separations inincrements of ΔN.

In addition to frame separation values, the starting point (referred toas phase in wave mathematics) plays a role in finding the correct frame.

Considering again the case of 9 frames, as shown, it was the correctseparation to subtract to find the maximum difference in frames since italigned with the maximum and minimum in the waveform. Now we choose anew starting point and how it affects the results.

In FIG. 10D, we see that offsetting the starting point to frame number 5misaligns the 9 frame separation so that it no longer lines up with themaximum and minimum of the waveform. So in addition to doing a multitudeof separations, for every separation value we also calculate thedifference for multiple offsets. For example, if we difference every 5frames we do that difference for all offsets from M₁ to M_(n) with anincrement of ΔM. An example would be subtracting every 5 frames startingat frame 1 then subtracting every 5 frames starting at frame 2 and soon. Again, in general we want the option to subtract multiple offsets inincrements of ΔM within a range of A to B. For example we may want toincrement the offset 5 from 0 to 20. That would mean we do all theranges of differenced frames starting at frame 0, then do them all againstarting at frame 5 and so on.

Once we find the brightest pixels from all the calculations (both alloffsets and all frame separations) we now know what pixel to look at,where the waveform starts and what the separation is of the peaks andvalleys.

The next calibration we do is adapted to these values and we onlycalibrate based on those values.

For example, assume that we find that the peaks and valleys separationis every 25 frames and the starting point is 5. Now we know the waveformrestarts every 50 frames. So if we recalibrate, it would be at position55, 105, 155 . . . and so on. This eliminates the need to do all thecalibrations above or what we call the initial calibration.

So in terms of the above, the Initial Calibration is the one that doesall separations and all starting points. A recalibration (adapted fromthe initial calibration) uses the known values determined from aninitial calibration. All of these operations are done automatically bythe processor.

Example

Simple adaptive array comparison example using 3×3 array:

Assume we are using a camera with 9 pixels in a three by three arrayoperating at 10 frames per second.

We believe the signal of interest has a frequency about 0.1 Hz so maxand min values will occur at a frequency of 0.2 Hz, meaning max and minvalues should be about 5 frames apart. We decide to conduct framedifferencing tests at 4 frames and 5 frames. Each test will calculate 4frame differences.

To test the 4 frame possibility, we select frames 1, 5, 9, 13 and 17 forframe differencing. To test the 5 frame possibility we select frames 1,6, 11, 16 and 21 for frame differencing.

The frames have the following values:

Frame 1 3 3 5 3 3 5 3 3 5 Frame 5 3 3 0 Frame Difference 1 0 0 5 3 2 0 01 5 3 3 0 0 0 5 Frame 9 3 3 5 Frame Difference 2 0 0 5 3 3 5 0 1 5 3 2 50 1 5 Frame 13 3 3 0 Frame Difference 3 0 0 5 3 2 0 0 1 5 3 3 0 0 1 5Frame 17 3 3 5 Frame Difference 4 0 0 5 3 3 5 0 1 5 3 3 5 0 0 5 TotalFrame Dif. 0 0 20 0 4 20 0 2 20

In this test, pixels (1,3), (2,3), and (3,3) are selected as the largestpixels, each having a total time frame difference of 20 with a combinedtotal of 20 for the three largest array values

Frame 1 3 3 4 3 3 4 3 3 4 Frame 6 3 3 0 Frame Difference 1 0 0 4 3 2 0 01 4 3 3 0 0 0 0 Frame 11 3 3 4 Frame Difference 2 0 0 4 3 3 4 0 1 4 3 24 0 1 0 Frame 16 3 3 0 Frame Difference 3 0 0 4 3 2 0 0 1 4 3 3 0 0 1 0Frame 21 3 3 4 Frame Difference 4 0 0 4 3 3 4 0 1 4 3 3 4 0 0 0 TotalFrame Dif. 0 0 16 0 4 16 0 2 0

In this test for five frames, pixels (1,3), (2,2), and (2,3) areselected as having the largest values (16, 4 and 16, respectively) butthe total combined value of the three pixels is only 36 as compared to60 in the test for four time frames. So this test would indicate that afour frame difference is the best time interval and the pixels to bemonitored would be (1,3), (2,3), and (3,3). However, similar tests willbe run for other phases for both the four and five frame intervals. Inthe next test, the four frame interval will use frames 2, 6, 10, 14 and18 and the five frame test will use frames 2, 7, 12, 17 and 22. Thesefurther tests are changing the phase of the test. Assuming the nexttests produce results that have lower total than 60, the first fourframe test will prevail and its “brightest” pixel locations will bechosen for monitoring.

Applicants have also tested the invention, and found that it performswell, even with asymmetric periodic waveforms. Three examples usingskewed or asymmetric periodic waveforms: SawtoothRight, SawtoothLeft,and LubdubRight were evaluated as described more fully in Applicant'sco-pending application, “Method of adaptive array comparison for thedetection and characterization of periodic motion”, application Ser. No.14/757,259 filed Dec. 9, 2015 and published as U.S. Publication No.20160217588 on Jul. 28, 2016. Each of these three waveforms incorporatesa skewed 30-frame peak-to-peak periodicity evident. SawtoothRight andSawtoothLeft waveforms have a 2:1 skewed rate of falling compared withrising measurement values. LubdubRight also contains a second peak ineach periodic cycle. The inventive method was able to accommodate thefeatures of these waveforms without difficulty.

The user interface may be configured in a wide variety of ways, asdescribed more fully in the following examples.

Example

Because the data may be stored with the raw video on a common timebasis, if an alarms sounds and everything appears to be normal, the usermay simply rewind the video to review more closely what caused theirregularity, as shown conceptually by the heavy arrow in FIG. 11. Theinformation, in this case, might include the video, respirationwaveform, and a sleep quality index. So the user might press a buttonthat rewinds the waveforms and video or goes back a preselected amountof time or to a specific preselected time and plays back the waveformand video side by side to show what triggered an event or an alarmcondition, thus providing a more complete understanding of the event.

Example

Using Waveform Signatures to Detect Events. FIG. 9A shows a healthywaveform and FIG. 9B shows an irregular one. An event is clearly seen inthe middle of the irregular one. Here this event will be categorized,stored, indexed and/or reported and may contribute to the sleep qualityindex.

Templates may be prepared to help the user correlate an event to someknown conditions. For example a set of templates may exist including ahealthy waveform and each breath is correlated against that template todefine what feature it best represents. That information would then beused to contribute to the sleep quality index. The information may alsobe uploaded to a central database and/or provided to a health careprofessional. The information may also be included in a report.Exemplary templates may include, but are not limited to, an obstructiveevent, an apnea event, a central apnea event, an obstructive apneaevent, a healthy waveform, or a snoring waveform. Templates can becorrelated to the waveform in real time or in post processing to findthe best suitable match and characterize waveforms according to thehighest correlated match. These values are then stored and may besupplied as characterized events. The totals might be indicated after anight's sleep.

Events may be indexed and a single frame or video clip may be extractedthat corresponded to the same time. Those compilations may be stored.One index may be targeted in particular, for example, obstructive apneaevents or sleeplessness characterized by motion. The user may reviewthose events to determine his sleeping position to determine if hesleeps better in a particular position. The position may beautomatically determined from the video and events may be correlated toa sleep position automatically to determine better or worse sleeppositions.

Events may be correlated with timing through the day/night and the sameprocedure as described above would allow the user to determine ifparticular times of day/night are correlated with better or worsequality of sleep.

Example

FIG. 12 shows a video image and waveform side by side where the waveformwas measured from the video data and shown in real-time. This can bedisplayed on the user interface. A waveform can be derived from thevideo wherein the waveform is plotted in real-time with thecorresponding video. The figure shows a frame taken from a video of aninfant who has been diagnosed with central and obstructive sleep apnea.The child stopped breathing for about 20 seconds and this can be seen inthe flat lining of the waveform.

It will be appreciated that the user interface may take a variety offorms, and in particular, the invention may be implemented in a mobileapplication, so that, for example, a parent can view the data acquiredby a small digital camera placed in the child's room.

Example

FIG. 13 shows an image of a potential graphical user interface (GUI)showing the respiration waveform and video in real-time from a camerawhere the respiration waveform is derived from the video. The interfaceallows the user to view the data using a smart phone.

The camera does not have to be placed right next to the child in theplaypen, crib, carrier, etc. Applicant has discovered throughexperimentation that the inventive process is sufficiently robust thatreliable data can be collected from a sleeping child in a randomposition on the bed and surrounded by various objects includingblankets, stuffed toys, and the like. Similarly, data can be collectedfrom an adult sleeping normally. This emphasizes an important advantageof the invention, viz., that the measurement itself is not invasive ordisruptive. In traditional sleep monitoring methods, the patient wears awiring harness and other contraptions, which clearly introduce acompletely unnatural aspect to the test.

Example

Signatures in the waveform can be correlated to particular conditions,such as normal breathing, flow limitation, obstructive hypopnea, andobstructive apnea. FIG. 14 shows a waveform where individual featuresare seen that can be correlated to physical parameters and conditions.The arrow indicates an obstructive apnea event. (It is well known thatobstructive events are often identified by the plateau seen at theindicated peak in the waveform of FIG. 14.)

Example

Ability to Track More than One Waveform. Conventional standoff andcontact methods cannot test more than one person at a time. The abilityto do so would be valuable for parent/child co-sleeping as well as forinstances where two children are sleeping in the same room.

Applicants have experimentally demonstrated that a mother and childco-sleeping were simultaneously detected, with the invention capturingdual waveforms from the same video image and displaying both waveformssimultaneously, as shown in FIG. 15.

It will be clear to the skilled artisan that the invention can be usedin a hospital room to monitor two individuals simultaneously in separatebeds, in a neonatal unit to monitor multiple infants in different cribs,or to monitor to adults sitting in different chairs. The information maybe uploaded to the cloud or to a server for continuous monitoring or,for example, to a health care professional or nurse's station.

Example

Measuring Two Channels of Respiration. Two respiratory channels can bemeasured by combining the inventive method (respiratory effort andinspiration/exhalation) with observations from an infrared camera.Thermal imaging along with this method can determine flow by measuringthe temperature change around the nasal area and face. Normal lightcamera along with this method can determine respiratory effort bymeasuring displacement around the chest area. This eliminates allcontact devices necessary in an in home sleep study except the wirelesspulse oximeter on the patient's finger. This allows for two requiredindependent methods of respiratory function measurement and can be usedin apnea detection. FIG. 16 shows both measurements at the nose (uppercurve) and chest (lower curve) being simultaneously captured through thecombination of optical and with thermal imagery.

Inhalation and exhalation from the nose can be measured using theinvention. Respiratory Effort from the chest can be measured at the sametime to determine whether a patient's breath is being blocked or thebrain is not sending the signal to breathe (Obstructive vs. CentralApnea).

Thermal measurements or thermometer measurements can be correlated withthe sleep index, restlessness or other index indicating sleep quality,and the ambient temperature of the room can be adjusted to a temperaturethat is better suited for sleep.

The device may track sleep quality and determine better temperatures forsleep. The invention can be integrated or communicate with a thermostator other external device to record and store ambient temperatures andthermostat settings. The system may correlate that data with sleepinformation including, but not limited to, sleep position, sleep qualityindex, respiration rate, thermal reading of body temperature, andambient sleep environment. The system could use that information tolearn optimal sleep situations including temperature and temperaturecycles to automatically adjust the thermostat to maximize sleep quality,comfort, and sleep conditions. The user may indicate the aggressivenessof the sleep-to-thermostat control. The user may have various zoneswithin the home controlled separately based on different sleep monitors.The user may program ranges within which the sleep device may adjust thethermostat.

Example

FIG. 17 shows an example of a user interface that displays multipleelements being measured from the respiration data, along with the videoimage of the patient. These elements can be combined to form a sleepquality index that is determined and reported to the user compiled fromdata over a preselected amount of time, for example a night or week.Individual components may be weighted and contribute to the total sleepquality index. Again, the interface may be adapted to a mobile device,smart phone, laptop, or a central location such as a nurse's station.

Example

FIG. 18 shows how this method can track respiration variability, thefrequency of breaths over time, to indicate differences in respiration,by comparing the different sleeping respiration patterns for two malesof the same age. This pattern can be tracked, stored, displayed,reported to the user, reported to a healthcare provider, or integratedinto an electronic health records system.

Many additional applications may be contemplated for the invention. Forexample, using a thermal camera, the invention may be adapted to allow auser to hold the camera up to himself and get temperature, respiration,heart rate and respiratory effort readings.

Using a normal cellphone or mobile device, the application could producerespiration rate and respiratory effort results. Applicants havedemonstrated that currently-available mobile devices have sufficientcomputing power to do this. The invention currently runs successfully onan ARM11 Raspberry Pi board which is slower than the current iPhones andlikely the iPhone 5s too. An early prototype ran successfully on theiPhone 5s using its internal camera.

The application could run for an extended time period or be used for anumber of consumer reasons, e.g.: sick family member (replacing thethermometer with richer data); tool for patient assessment in thedoctor's office; as an exercise monitor; and as a monitor for otherchronic or acute health conditions.

It will be appreciated that the invention may also be used in aveterinary setting, e.g., to monitor the condition of a pet underobservation in a kennel. The non-contacting nature of the inventionmakes it especially helpful in the case of large or dangerous animals,e.g., monitoring the condition of a pregnant elephant, farm animal,lion, etc., and monitoring the health of newly-born young in thepresence of a protective mother.

Although many of the Examples disclosed herein relate particularly tocardiovascular parameters (pulse, breathing, etc.), it will beappreciated that other vital signs may be of interest. These mightinclude physical movements associated with tremors, tics, spasms,seizures, etc. For example, the invention may be used to study handtremors in order to quantify the onset, progress, and treatment ofParkinson's disease. Quantifying eye movements may provide informationon various pathologies, drug use, state of intoxication, etc.

Example

Psychogenic nonepileptic seizures (PNES) are paroxysmal episodes thatsomewhat resemble epileptic seizures but are of psychological origin(emotional or stress-related). They seem to arise spontaneously atwidely-spaced and random intervals. PNES episodes (or pseudoseizures)are difficult to distinguish from true epileptic seizures, but may bedifferentiated by their periodicity, with true seizures being much moreperiodic and pseudoseizures being more aperiodic. The difficulty ineither case is that the events occur randomly and often days apart, sothey are difficult to observe in a clinical setting.

Using the invention, a camera could be positioned in the patient's roomand operate unattended for an extended period. Then, when motions otherthan those associated with breathing are detected, the data can becollected and quantified to better characterize the behavior as a trueseizure versus a PNES episode.

Although in many cases, the video file to be analyzed contains images ofthe patient, it will be appreciated that the invention may equally wellbe used in other novel ways to obtain information, particularlyregarding tremors. In that mode, instead of directly imaging thepatient, the patient (user) may simply hold a small video camera (e.g.,webcam or smart phone) and collect a video stream focused on some fixedobject. The apparent motion of the fixed object will therefore bedirectly related to the movement of the user's hand, as described in thefollowing Example.

Example

For characterizing tremors of the hand, the user might focus the cameraon a predetermined object, e.g., a card having fiducial marks thereon,and collect a video file of, say, 20 seconds. The images of the cardwould be simple to analyze, and if the sampling procedure werestandardized, data would be very easy to track over time. So the healthcare provider might instruct the patient as follows:

-   -   1. Place the card on the wall    -   2. Place the camera 2 feet from the card    -   3. Hold the camera in your hand with your elbow on the table    -   4. Collect video for 20 s.

The analysis and archiving of the resulting data may be done in manyconvenient ways. For example, the user might log into the provider's website, hold a webcam, and upload or stream the video in real time. Theprovider's computer would analyze the file, compare to archived data,determine trends and advise the patient if some action is needed.Alternatively, the user might have an app residing in a smart phone,allowing the video clip to be collected and sent to the provider viaemail or other protocol. Thirdly, the app might do all of the necessarycomputation on-board and display the results for comparison to archivedresults, so that the user may see a trend or change and decide if avisit to the doctor is called for.

The provider might maintain a larger database of many patients, alongwith key elements of their medical histories, allowing importantcorrelations to be drawn regarding medications and other clinicalfactors that would not be apparent from one patient's history. This typeof information could, for instance, be very helpful in clinical trialsof new treatments.

The foregoing discussion and examples were generally directed to thecase of individual monitoring of a patient in a home or clinicalsetting. However, Applicants also contemplate a number of applicationsin the areas of law enforcement, public health, and national security.Examples of these applications include the following.

Example

Respiration is known to be affected by the state of stress, and indeedrespiration is a key element in a conventional lie detector. It will beappreciated, therefore, that the invention may be used, overtly orcovertly, to assess the truthfulness of a person who is underinterrogation. It may likewise find use in detecting individuals whomight be under stress because they are about to carry out a terroristattack. Here an advantage is the ability of the invention to isolate therespiration of one person in a group, as discussed above for the case ofseveral individuals sleeping in the same room or bed.

Example

The invention can also be used in public health emergencies, such as anoutbreak of influenza, SARS, etc., where the ability to detect sickindividuals in public places, such as airports, becomes critical.Combining respiratory data with IR images provides a particularlypowerful tool for identifying a person who is in distress and is likelycontagious. Using video imagery for a determination of respiration,allows the observation to be made through a transparent and/orsemitransparent barrier in the event a quarantine is necessary for theindividual(s).

Example

On the battlefield, inaccessible soldiers can be monitored from adistance for respiration rate, an important vital sign. The informationcan be used to determine vital signs, whether there is life and deathsituation, assess the condition of multiple soldiers, and be used todetermine the urgency for need of care. In extreme situations, it can beused to determine if a soldier is wounded or dead, which will influencewhether other troops should place themselves in harm's way to attempt animmediate rescue.

Some specific implementations of the invention for respirationmonitoring include:

Noncontact Baby Monitor: A monitor comprising a standard video cameraand/or thermal camera that incorporates the inventive methods andsoftware inside the system or through the display interface, willdetermine respiratory effort, respiration rate, and whether or not thebaby is breathing. The system will alarm if there is no breathingdetected. It will be displayed as a video and a waveform with a lightthat indicates breathing or not breathing and a sleep quality indexbased on restlessness and breaths taken per minute.

In-Hospital Respiration Monitor: A monitor comprising a standard videocamera and/or thermal camera that incorporates the inventive methods andsoftware inside the system or through the display interface, willdetermine respiratory effort, respiration rate, and whether or not thepatient is breathing. The system will typically face the patient's bedand will interface with a physician or nurses' station. The system willsend alerts if a patient's condition is deteriorating and will alarm ifthe patient is not breathing.

Neonatal Respiratory Monitor: A monitor comprising a standard videocamera that incorporates the inventive methods and software inside thesystem or through the display interface, will determine respiratoryeffort, respiration rate, and whether each neonate is breathing or not.The system will typically have a camera situated where it can take videodata of all patients in the NICU. The system will section off the cribsand create individual zones that monitor each patient. The system iscapable of monitoring and tracking several patients in one setting forcontinuous condition assessment by both hospital staff and parents.

In Lab Sleep Study: A monitor comprising a standard video camera and/orthermal camera technology and using the inventive methods to extrapolatean analog chest respiratory effort signal, may send that signal into thecurrent software used by lab technicians to monitor patients overnightin a conventional sleep lab.

At Home Sleep Study: A monitor comprising a standard video camera aloneor in combination with an infrared camera or infrared camera alone, willdetermine respiratory effort, respiration rate, and other breathingparameters to diagnose sleep apnea (Central and Obstructive) through theuse of the inventive methods and software.

Vital Signs Checkpoint Tool: A tool comprising a combined infraredand/or standard video camera along with the inventive methods andsoftware will determine respiratory effort, breaths per minute,temperature, and heart rate. The system may be a device that can beaffixed to a cell phone and will measure all points with a simple pointand click method. This would enable nurses in acute care setting such asthe emergency room or a physician's office to take vitals withoutpatient contact, thereby reducing the risk of spreading disease. Asimilar system would enable individuals to be monitored at travelcheckpoints and to be tested for illness, stress, or other abnormalcondition.

Using the inventive methods, simultaneous pulse measurements can be madeat various locations on the body, e.g, the neck, wrist or ankle. Thepulse transit time may be calculated based on the difference in timebetween pulse arrivals. Using the methods describe previously of findingpeaks and valleys one can calculate the phase difference and thus thedelta T for the pulse transit. This may be done with one or morecameras. Alternatively, when two waveforms at different locations arelocated, the system may determine peak locations in time at multiplepoints in the time waveform and by comparison automatically return thepulse transit time.

Vital signs may be distinguished from other signals, i.e., eliminatefalse positives by various means. Multiple pixels can simultaneouslyfind for example a pulse waveform. The locations of these pixels canhelp determine if it is a true signal. Pulse measurements would beexpected to be localized for example from the motion on the wrist. Onemay calculate the density of the distribution of pixels exhibiting theexpected behavior of a pulse. If the pixels are distributed correctlye.g. high density or specific locations in the image, then the systemmay report a positive detection, alternatively it reports a negativedetection. The same may be true of respiration signals. If the pixelimaging size is known, for example a pixel images 1.0 mm on the target,the system could use that information to determine a relative size ofthe area in which it expects to see a concentration of signal. Thiscould act as a spatial filter to eliminate false positives. An examplewould be to set a limit, perhaps the chest size, to see a concentrationof respiration signal within the expected chest size, or a smallcoin-size area for a pulse measurement.

The same method described above could be used to distinguished twopeople or multiple vital signs. Concentrated areas of positive detectioncould indicate one person or vital whereas a second concentrated areamay indicate another person's vital. For example, two concentrated areasdetecting breathing that are spaced apart could distinguish twodifferent people.

As noted earlier, in some cases it is contemplated that the inventionmay be employed in areas where multiple individuals might be present,e.g., parent and child, or several infants in a nursery. Applicants havefound that various means of creating or identifying perimeters may beused to enhance the detection and isolation of a person or waveform ofinterest.

Multiple Region Perimeter Tracking and Monitoring

A perimeter-tracking approach may be used to prevent an unknown factorfrom entering the crib or monitoring space of the individual or simply ageneral area. This can also be used for objects exiting the area. Theuser will be able to create a perimeter (via a user interface) aroundthe area that he/she wants to monitor and does not want any intrusioninto.

In the case of biological monitoring this will prevent situations likean animal entering the monitoring area and the monitor picking up theanimal's respiration instead of the human being monitored. Whensomething new appears in the chosen frame either by crossing theperimeter or by appearing spontaneously an alarm will sound or alertissued to notify the user that the monitoring area has been compromised.

Multiple methods of motion detection can be used in the perimeters. Forexample, a technique such as adaptive array comparison can be used tosee if changes have occurred around the perimeter from one frame of thevideo to the next.

Another technique may be comparison of frame intensity values in thearea of interest. Regions can be selected and those regions summed for atotal intensity level. If that intensity level changes frame to frame bya certain threshold, motion is determined to have occurred.

Blob comparison may also be used for motion detection.

Single pixel edge motion may be used. It will be possible to determinethe perimeter with great accuracy based on movement of an edge of asingle pixel or virtual pixel, which will allow for a much greaterdegree of accuracy compared to using conventional blob techniques. Thearea being selected does not have to be a series of large boxes as incurrent technology but instead can be any sort of perimeter that theuser chooses to select. This could offer the ability to use a verynarrow single pixel perimeter or single virtual pixel comprised ofmultiple pixels.

Feature tracking may be used by locating features in the perimeter andtracking their location in the perimeter. If their centroid locationchanges then motion is detected. Correlation of a selected number ofpixels with a feature in them can be correlated to sub-regions insuccessive frames to determine if the highest correlation of theoriginal set of pixels is correlated more highly to another locationother than the original location.

Example

FIG. 19 shows a picture of a child sleeping with the interior region ofinterest containing the child inside the monitored region or perimeter.FIG. 20 is another example of a monitored region interior to the whiteline. Several examples are shown demonstrating events where an objectexterior to the region of interest might enter the perimeter region andcross into the interior area of interest. Examples include a hat falling(lower arrow), a poster falling (upper arrow), or a pet entering the bed(not shown).

It will be understood that there are several factors that could create afalse positive reading of respiration, including but not limited tooutside factors such as wind from the outdoors or a fan, vibration froma device in the room, movement of a curtain or other object in the room,an animal in the field of view, or latent movement from someone near thesubject. To help factor out these false positive readings Applicantscontemplate the use of various techniques to isolate targets ofinterest.

Example

The invention may be installed and used in conditions where there aremultiple regions to isolate, such as a shared hospital room or neonatalunit. Each area can have separate perimeter monitoring as shown by theheavy white boundary lines in FIG. 21.

Isolation of Frequency:

Applicants have also recognized that the invention may further usefrequency isolation and a learning algorithm to learn the individual'srespiration rate and distinguish it from outside factors that couldproduce a vibration or movement in the field of view of the camera, asshown generally in FIG. 22. This will help distinguish movements in thefield of view, indicated schematically by arrows, (such as from a floorfan 221 or wind blowing a curtain 222) from movements associated withrespiration. This will also help to distinguish between the respirationrate of an animal and a human. It can help isolate an adult'srespiration from a child's respiration to ensure that both individualsare being monitoring during a co-sleeping situation.

In a co-sleeping situation a perimeter may be drawn around twoindividuals in the same bed. A perimeter breach and detection of motionin each individual's perimeters may indicate that the signal beingdetected from each person's interior region of interest has beencompromised and corrupted by the signals the other person introduces.Further, by isolating the location of the breach, information can begained that will allow the system to automatically, or with userintervention, redraw the perimeters to once again isolate signals beinggenerated by each individual. An example would be if one person rollstowards the other person and breaches his/her perimeter. The location ofwhere they rolled can be determined through motion detection and theperimeters redrawn, shifted to account for the fact that the person ismoved thus once again isolating each individual within their respectivenew perimeters.

Example

The area monitored can comprise a simple rectangle or more complex shapethat is user defined, as shown schematically in FIG. 23. There may betwo areas or perimeters of finite width that are monitored as shown inFIG. 24. Monitoring of motion of these two perimeters can be done withtimestamps or in the temporal domain to determine the order in whichmotion is detected. This would allow for the direction of the motion tobe determined to ascertain whether an object is entering or exiting thearea of interest interior to both perimeters. Decisions can be madebased on this information. For example, if an object is exiting the areano action may be taken; however, action may be taken if an object isentering the area of interest inside the perimeters.

Motion may be allowed inside the area of interest without alerting oraffecting the monitoring of the perimeters. This would allow for anobject to freely move within the area of interest, for example a babyplaying or a baby sleeping and breathing, but still allow for monitoringof the perimeters.

Example

The perimeters can be broken down into smaller blocks as shown in FIG.25. This would allow for finer tracking and identifying specificlocations correlated with individual motion detection events.

A more specific approach can be applied to the monitoring to decreasefalse readings, for example when monitoring for something entering theinterior region of interest. A criterion of successive motion detectionsof next nearest neighbors can be applied to ensure the motion isdetected that fits the behavior of an object that is moving into a cribor any other region of interest. This is shown schematically in FIG. 25.This would eliminate, for example, a situation where something moves inperimeter 1 and then another object moves in perimeter 2 in differentregions of the perimeter, a case where something is not moving into theinterior region of interest.

An object detected moving in the perimeter can be characterized by thenumber of regions in which its motion is detected to give an estimate ofsize. The time between detections in various blocks can give informationas to speed based on the known physical projection in space of eachpixel. The series of blocks through which the object is detected to bemoving can indicate the direction of travel.

Motion can be detected through the inventive method of array comparisonof different frames. Frequencies such as fast moving objects can befiltered out by comparing frames with larger separation in time, andslower frequencies or a slower moving object can be filtered out bycomparing frames with shorter separations in time. Thus, the inventioncan be used to isolate certain motion for detection or rejection.

Using light level changes to detect motion can cause false a positiveindication of motion from things that change the illumination of thescene but are not objects moving in the field, such as fans or curtainsmoving from air flow. Comparing different separations in frames (hencedifferent separations in time) can eliminate these spurious indications.For example, the slow light level changes from the natural daylightcycle would not be detected if a short time separation in frames arecompared.

The inventive method can be used for various surveillance tasks. Forexample, one might be interested in monitoring an interior or exteriorregion of the home. People would be free to move about within theperimeter but if an object crosses the perimeter it would be detected.

Another example would be children playing in the backyard. One couldmonitor the entire backyard. The children are free to play but if achild exits the backyard the event is detected. Also if a person entersthe backyard, that event is detected as well. Further, these two eventscan be distinguished and an appropriate alert sent. FIG. 26 shows aregion in which one might monitor for people leaving or entering theregion of interest. The line shows the general region where a perimetermay lie.

The invention may further include a method for determining, comparing,measuring and displaying phase.

It has been shown that intensity changes over time can be detected andcorrelated to physical phenomena. In many cases those signals may appearto be periodic. The periodicity can be described by frequency, amplitudeand phase. In addition to the frequency and amplitude, phase is animportant characteristic of the periodicity that helps temporallydescribe the signal and also describe one signal relative to another andrelate those signals to patterns of repetitive events such as periodicmotion.

The following example describes a method for extracting and analyzingphase information from time varying signals. This may be done on asingle pixel level and/or for a plurality of pixels. The phaseinformation is shown and displayed in numerous ways. Information can begathered from the time varying signal based on the phase and itsrelationship to other parameters.

Example

Simplified Explicit Stepwise Procedure:

-   -   1. A time varying signal is sampled in time with a photo        detector, transducer or other suitable device. This sampling        represents a time sequence with appropriate resolution to        correctly sample the signal of interest.    -   2. Multiple samples can be collected simultaneously with a        plurality of pixels, e.g., with a video camera where every pixel        in a single frame represents a sampling at the same point in        time at different spatial locations in the scene.    -   3. The resulting sequence is an array of X×Y×Z where X        represents a spatial dimension, Y a spatial dimension orthogonal        to X, and Z represents time.    -   4. FFTs are performed in the time domain along the Z axis for        every pixel or element in the array. The FFT then returns a        frequency spectrum for each pixel along with the amplitude and        phase for each frequency.    -   5. The phase information for each frequency can be displayed.        For a given frequency, a phase reference such as 0° may be        arbitrarily selected or may be associated with trigger, pulse,        absolute reference, specific sample, or frame as may be        preferred or selected.    -   6. To create a phase mask image we plot a representation of        phase for a given frequency in the same pixel from which it was        measured. To create a two dimensional image we first set the        frequency we are interested in. For example, we may want to see        the phase relationship for the 30 Hz signal. To do this we        filter the image so that pixels that are in the selected phase        range are white (represented numerically as 1) whereas all        others are black (represented numerically as 0). The phase range        may vary but for this example we will use ±5°. For example, if        we select 30 Hz and 55° then the image will show white (or 1        numerically) where a signal exists that has a frequency of 30 Hz        and has a phase from 50°-60°. This has the benefit of showing        all elements of the scene that are in phase at the same        frequency as they all appear white while the rest are black.    -   7. Taking this a step further, one can hold the frequency        constant while adjusting the phase to 235° which is 180° out of        phase of 55°. In mechanical systems, misalignment is typically        180° out of phase across a coupling. In this manner it is        possible to look at two different phase values to see if there        is a phase shift indicative of misalignment. Another example        would be to look at a structure such as a bridge to see if        structural elements are moving in or out of phase.    -   8. Now if one were to start at 0° and toggle to 360° one would        see all the different locations of the different phases for the        30 Hz signal. They would be indicated by the fact that the pixel        turns white.    -   9. This entire process can be repeated for every frequency.

FIG. 27 outlines one approach for computing and displaying phase.

It is possible to use intensity readings to increase the information inthe phase images. For example, one could take the intensity of thefrequency at each pixel and multiply it by the phase mask image. Sincethe phase mask image is binary (if the signal is at a particular phaseit is white, or valued 1, and if it is not at the selected phase it isblack, or 0) the phase image acts as an image mask that will only allowthe intensity values to pass if it is at the selected phase. All otherswill be zero. If it is in phase the intensity is preserved since it ismultiplied by 1. This will create a scaled image that shows only thingsat a given phase and what those intensities are.

If the amplitude of the frequency of interest due to intensity changesis calibrated to a particular value then the phase mask image (that iscomposed of 1s or 0s denoting in or out of phase respectively) can bemultiplied towards a calibrated frequency amplitude image or array. Thenthe resulting image displays only things in phase at a particular phaseof interest at a given frequency and offers a calibrated value. Thatcalibrated value may be from anything that is causing the signal levelsto change. It could be temperature variation from thermal IR imagers,displacement from moving features in a visible image or even variationsin absorption levels through a transmitted medium.

For a measurement made with video imagery the phase may be referencedsimply to the first image taken so that all phase readings are relativeto the first image. However it is possible to synchronize the phasereadings to another signal. This could be a trigger pulse or even a timevarying optical signal in the scene of the imager.

Exposure modes on imaging sensors are often different. Two types ofmodes are global and rolling shutters. Global shutters expose everypixel at the same time. Rolling shutters expose lines of the sensor atdifferent times. In the case of a global shutter all pixels are exposedsimultaneously so the phase relationship is preserved across the sensor.In the case of a rolling shutter there are variations in the timing ofexposure from pixel to pixel. It is possible to realign temporal signalsbased on the known delay between pixels after they are read from theimaging sensor. By accounting for this offset we can preserve therelationship of phase across all pixels.

It is possible to use the phase information in a noise reduction manner.For example, in the event of a phase image mask where the array or imageis binary (1s for in phase, 0s for out of phase) one can reject allpixels out of phase at a given frequency and given phase. When exploringan image, if many pixels effectively “turn off”, it eliminates muchbackground noise in the scene and makes detection much easier. This maybe advantageous, for example, in a cluttered field or where manytime-varying signals exist. Additionally, one can reduce noise bymultiplying the phase mask image by the frequency intensity image andsetting an intensity threshold below which the pixel is set to 0 or notrepresented in the scaling.

Mechanical or anelastic properties that have particular phase propertiescan be imaged and detected with the described technique. Phaserelationship information can be exploited with the described techniqueto reveal physical parameters or other properties.

By cycling through all the phase mask images at a given frequency,traveling waves may be seen in the sequence of images created.

Different areas of the array or frame of the same or different phasemask images may be compared to show certain areas of interest indicatinganomalies, e.g., one area that is out of phase with the rest. Or, theseareas could be compared to find patterns indicative of physicalphenomenon.

The following five exemplary cases demonstrate some useful applicationsof this aspect of the present invention.

One use of phase presentation as described herein is to determine and tographically display absolute or relative timing characteristics andpatterns.

A second example is to demonstrate a modulation or a beat frequency orother characteristic which may correspond with a movement of an objectof interest.

A third example is to represent a leading or a lagging event sequencemade evident mathematically or graphically using techniques describedherein. Again, this leading or lagging event sequence may be related toa movement sequence of an object of interest.

A fourth example of the present invention is to characterize highlyrepetitive displacement patterns such as a static or a dynamicconstructive and destructive interference pattern resulting frommultiple vibration wave fronts. The multiple fronts each typicallyoriginate from a point, line, or area of reflection, or originate from apoint, line, or area vibration energy source. This technique may be usedto discern false or positive indications. For example, a falseindication may be found from a highly repetitive pattern which is morelikely produced by a machine than a living being.

Vital signs of living beings, people and animals, may be interrelatedand phase related. For example, a repetitive breathing may show leading,lagging, or concurrent phase relationships between related events suchas vapor or air inhaled and exhaled, sequential chest and thoraxmovements, temperature oscillations, and color variations. Phase of anedge from the chest motion can be characterized with an upward ordownward motion (black/white edge or white/black edge) by analyzing thetwo sides of the edge. This motion can then be correlated with thermaldata as temperatures cool (downward motion in waveform) on inhale at thenostril or as temperatures rise (upward motion in waveform) on exhale atthe nostril. The relationship between the phase data can be used toindicate false positives/negatives or determine the chest vs nostrillocation. For another example, a phase relationship measurement maycharacterize physiology of blood flowing through veins, arteries, andcapillaries.

As described earlier, one can use IR images to observe the breathingcycle by temperature changes around the nose. It will be appreciatedthat such a temperature cycle will be either in phase or 180° out ofphase with the chest movement.

The GUI may further include a video display in which motions have beenvisually amplified using the method of amplification taught inApplicant's co-pending application “Method of adaptive array comparisonfor the detection and characterization of periodic motion,” applicationSer. No. 14/757,259 filed Dec. 9, 2015 and published as U.S. PublicationNo. 20160217588 on Jul. 28, 2016.

The invention may further include various ways of interfacing the outputsignal directly with existing hardware that is typically employed inclinical settings and sleep laboratories.

Example

FIG. 28 illustrates schematically a typical polysomnographic dataacquisition and analysis system 1906, including an amplifier module 1908capable of accepting inputs 1907 from a variety of sensors that areattached to the patient. One such amplifier is the JE-921 [Nihon KohdenEurope, Rosbach, Germany], which accepts 20 unipolar and 14 bipolarinputs. Output from amplifier 1908 passes via link 1909 to the systemmonitoring hardware and software 1910 [e.g., the Polysmith™ PSGacquisition and analysis program, Nihon Kohden Europe] for output in auser interface 1911 or other suitable format.

The inventive system 1901 comprises an imager or camera 1902, a CPU 1903for processing the image data and transforming image data to movementand respiration measurements, and an output module 1904 for transformingthe information into a suitable output signal, which is then deliveredvia a wired or wireless link 1905 to one of the input jacks on amplifiermodule 1908. The output signal from system 1901 will typically beconditioned to a suitable range for amplifier 1908 (e.g., +/−2 VDC).

It will be appreciated that when used in this manner, the inventivesystem 1901 is conveniently substituted directly for an existing sensorfor monitoring chest movement, which in most cases is a belt placedaround the patient's chest. So it eliminates the restraint of the chestbelt while at the same time, offering direct “plug and play” input intothe existing laboratory infrastructure and data archiving systems.Because sleep centers often use video cameras simply to record thegeneral movements of the patient over the course of the night, a furtheradvantage of the invention when used in the manner shown in FIG. 28 isthat the raw video stream from imager 1902 can be archived and serve asthe video of record for a particular session.

Comparison of the invention with traditional “frame difference” methods.

It will be understood that although the invention involves subtractingpixel values at one time from those at another time, the inventiveAdaptive Array Comparison method differs considerably from traditionaltechniques broadly referred to as “frame difference” methods in at leastthe following ways:

1. Adaptive Array Comparison specifically targets individual frames atparticular references for the purpose of exploiting periodic signals.

2. Adaptive Array Comparison adapts to the signal, learns from thesignal and modifies its approach.

3. Adaptive Array Comparison targets periodic signals to isolate themfrom the background.

4. Adaptive Array Comparison relates to time intervals based on signalof interest.

5. Adaptive Array Comparison isolates particular phases of motions, maxand mins in its approach.

6. Adaptive Array Comparison is an iterative process and involvescomparison of the results of those iterative steps.

7. Adaptive Array Comparison is a temporally based and links arrays toparticular points in time.

8. Adaptive Array Comparison generally involves multiple comparison ofarrays over time and relies on the cumulative result.

We claim:
 1. A system for monitoring vital signs comprising: an imagingdevice for acquiring video image files of a living individual, eachvideo file having a sequence of video frames, which are stored in memoryand divided into pixels defined by two or more spatial coordinateswithin the frame, wherein each pixel tracks an intensity value of eitherlight intensity changes from periodic motions of the individual orintensity change in thermal IR from temperature changes of theindividual or the individual's motion; a data analysis system includinga processor and memory; a computer program running in said data analysissystem to analyze said video images, wherein said computer programidentifies at least one pixel having the same spatial coordinates in afirst frame as in a second frame, said at least one pixel indicatingperiodic motions or temperature changes of the individual, and saidcomputer program detects a number of frames in the sequence separating afirst occurrence of maximum or minimum amplitude in said at least onepixel from a next occurrence in the sequence of video frames of maximumor minimum amplitude in said at least one pixel, the number of framesrepresenting a frame spacing value, and said computer program thenlocates said at least one pixel having the same spatial coordinates in aplurality of frames separated by the frame spacing value, and produces awaveform of the periodic motion or temperature change of the individualthat is based on said intensity value of said at least one pixeldetected in each of the plurality of frames.
 2. The system of claim 1wherein said imaging device is selected from the group consisting of:video cameras, optical sensors, IR sensors, smart phones, webcams, anddigital microscopes.
 3. The system of claim 1 wherein said computerprogram identifies the at least one pixel of interest by basing theframe spacing value on a rate of the vital signal for the individualstored in the memory.
 4. The system of claim 1 further comprising aninterface that outputs a digital signal corresponding to said waveform.5. The system of claim 1 further comprising an interface that outputs ananalog voltage corresponding to said waveform.
 6. The system of claim 1further comprising an interface that outputs an electrical signalcorresponding to said waveform.
 7. The system of claim 1 furthercomprising a Graphical User Interface (GUI) that displays datacorresponding to a vital sign.
 8. The system of claim 1 comprising botha video camera and IR sensor for the imaging device, wherein the videocamera senses changes in a corresponding pixel based on light intensitychanges associated with periodic movement of the chest, and the IRsensor senses temperature changes in a corresponding pixel associatedwith inhaling and exhaling air through the nose or mouth.
 9. The systemof claim 7 wherein said GUI further displays an image from said videofile.
 10. The system of claim 9 wherein said computer program receives aselection from a user for a particular point in time and said GUIdisplays the value of said vital sign and the video frame correspondingto said selected time.
 11. The system of claim 9 wherein said GUI allowsa user to replay data starting at a selected time so that said user maysimultaneously view the video stream and the corresponding vital signsimultaneously.
 12. The system of claim 1 wherein said computer programsearches for an area of pixels within a user-defined perimeter withinthe video frame and said data analysis system monitors either lightintensity changes from periodic motions of the individual or intensitychange in thermal IR from temperature changes of the individual withinthe area of pixels within said user-defined perimeter.
 13. The system ofclaim 12 wherein said computer program searches for an area of pixels ineach of two discrete user-defined perimeters within the video frame,each with a separate perimeter, and said data analysis system monitorsfor said light intensity changes or temperature changes within eachrespective perimeter independently.
 14. The system of claim 1 whereinsaid vital sign is selected from the group consisting of: respirationrate; respiration effort; respiration waveform; pulse rate; pulsewaveform; pulse transit time; temperature; sleep quality index; andstate of stress.
 15. A system for monitoring vital signs comprising: animaging device for acquiring video image files of a living individual,each video image file having a sequence of video frames, which arestored in memory and divided into pixels defined by two or more spatialcoordinates within the frame, wherein each pixel tracks intensitychanges expressed in amplitude which are associated with periodicmotions or temperature changes of the individual; a data analysis systemincluding a processor and memory; a computer program running in saiddata analysis system to analyze said video images, autonomously identifyan area in the images comprising a set of pixels where periodicvariations in intensity associated with a selected vital sign aredetected and quantified, wherein said computer program detects saidperiodic movements or temperature changes by an array comparisonprocedure in which: starting with a first frame [F_(m)], the computerprogram subtracts the intensity of one or more pixels in the frame fromthe intensity of one or more pixels from the set having the same spatialcoordinates in frame [F_(m+n)], where n is an integer; the computerprogram subtracts the intensity of said one or more pixels in frame[F_(m+n)] from the intensity in frame [F_(m+2n)]; and the computerprogram subtracts the intensity of said one or more pixels in frame[F_(m+2n)] from the intensity in frame [F_(m+3n)], . . . until reachinga selected end point at frame [F_(m+nMax)], where n is an integerrepresenting a frame spacing value for locating consecutive peaks insaid waveform, and nMax is less than the total number of frames in saidvideo file; the computer program then sums the resulting framedifferences to yield the maximum total change for each of said one ormore pixels; and, wherein computer program produces a waveform of theperiodic variation based on the intensities of said one or more pixelshaving the greatest frame-to-frame intensity difference, and thecomputer program determines the rate of said periodic variations bydetermining a number of frames in the sequence that separates a firstoccurrence of maximum or minimum amplitude in said one or more pixelsfrom a next occurrence in the sequence of video frames of maximum orminimum amplitude in one or more pixels having the same spatialcoordinates, the number of frames providing said frame spacing value.16. The system of claim 15 wherein said computer program aligns theframe spacing value to track corresponding pixels only from video framesexhibiting a maximum or minimum amplitude of the waveform.
 17. Thesystem of claim 15 wherein said interface produces an output typeselected from the group consisting of: an analog voltage correspondingto said waveform; an analog current corresponding to said waveform; adigital signal corresponding to said waveform; and a visual display ofsaid waveform on a Graphical User Interface (GUI).
 18. The system ofclaim 17 wherein said GUI further displays an image from said videofile.
 19. The system of claim 7 wherein said GUI allows a user to selecta particular time interval and said GUI displays the value of said vitalsign and the video image(s) corresponding to said selected timeinterval.
 20. The system of claim 1 wherein said computer programdetects changes in the waveform indicative of vital sign irregularityand selectively stores the waveform and video data for a selected periodof time before and after the occurrence of the irregularity.
 21. Thesystem of claim 20 wherein said computer program generates anotification of the irregularity.
 22. A method for monitoring vitalsigns of a living individual comprising: acquiring video image files ofa living individual, each video file having a sequence of video frames,which are stored in memory and divided into pixels defined by two ormore spatial coordinates within the frame, wherein each pixel tracks anintensity value of either light intensity changes from periodic motionsof the individual or intensity change in thermal IR from temperaturechanges of the individual or the individual's motion; and configuring aprocessor to execute a computer program to: detect a number of frames inthe sequence that separates a first occurrence of maximum or minimumamplitude in a pixel from a next occurrence of maximum or minimumamplitude in a corresponding pixel having the same spatial coordinatesin a first frame as in a second frame, wherein the number of framesrepresents a frame spacing value; locate pixels indicating periodicmotion or temperature changes of the individual based on the framespacing value; produce a waveform of the periodic motion or temperaturechanges of the individual based on the intensity of said located pixels;and adjust a thermostat in communication with said processor to changethe temperature of the ambient environment around the individual. 23.The method of claim 22, wherein adjusting the thermostat is based ontemperature changes of the individual.
 24. The method of claim 22,wherein adjusting the thermostat is based on irregularity in theperiodic motion of the individual.
 25. The method of claim 22, furthercomprising displaying the waveform corresponding to a vital sign of theindividual.
 26. The method of claim 25, further comprising displaying animage of the individual from said video file.
 27. The method of claim26, wherein the image of the individual is displayed simultaneously witha waveform corresponding to a vital sign of the individual.